Delivery of synchronised soundtracks for electronic media content

ABSTRACT

A method and system for streaming a soundtrack from a server to a remote user device for a reader of electronic media content. The soundtrack is defined by multiple audio regions. Each audio region defined by an audio track for playback in the audio region, a start position in the electronic media content corresponding to where the playback of the audio region is to begin, and a stop position in the electronic media content corresponding to where the playback of the audio region is to cease. The streaming of the soundtrack is based on control data generated by the remote user device.

PRIORITY CLAIM

This application is a continuation of and claims the benefit of priorityto U.S. patent application Ser. No. 14/332,521, filed Jul. 16, 2014,entitled “Delivery of Synchronised Soundtracks for Electronic MediaContent,” which claims the benefit of priority of U.S. ProvisionalPatent Application Ser. No. 61/847,266, filed on Jul. 17, 2013, thebenefit of priority of each of which is claimed hereby, and which isincorporated by reference herein in its entirety.

FIELD OF THE INVENTION

The invention relates to a system and method for delivery ofsynchronised soundtracks for electronic media content, such as but notlimited to text only or multimedia content comprising text presented incombination with imagery, videos or animations. The playback or deliveryof the soundtrack is synchronised with the individual reading speed ofthe user.

BACKGROUND TO THE INVENTION

To enhance the reading experience, various systems have been proposed inwhich audio soundtracks relating to the content of text, such as ane-book, are produced and played to the user while they read the text.Some of these systems aim to synchronise the playback of the soundtrackwith the user's reading speed.

One such system is described in the applicant's co-pending internationalPCT patent application publication WO2013/015694, the contents of whichis hereby incorporated by reference in its entirety. In this system, asoundtrack for electronic text, such as an e-book, is produced thatcomprises audio tracks of different types that are configured to playconcurrently and/or sequentially while the user is reading theelectronic text. Playback of the soundtrack is synchronised with thereading speed of the user. The soundtrack may comprise a combination ofdifferent audio types, including ambient noise, sound effects, andbackground music that are configured to playback at a specific point orover specific portions of the electronic text in a synchronised manneras the user is reading those portions of the electronic text. Thesoundtrack in WO2013/015694 comprises multiple audio regions that aredesignated to playback during designated text portions of the electronictext. Each audio region comprises audio data defining the audio track toplayback, the start position in the electronic text at which the audioregion is to commence playback, and a stop position in the electronictext at which the audio region is to cease playback. In one embodimentdescribed, the electronic text is indexed based on the number of wordsand the start and stop positions of the audio regions relate to a startand stop word.

WO2013/015694 describes various user playback systems for displaying theelectronic text and controlling playback of the associated soundtrack.The user playback system may be implemented on various user hardwaredevices, including portable consumer electronic devices such assmartphones, tablets and dedicated e-reader devices or personalcomputers, whether desktop or laptop. The soundtrack data file or files,including all the audio tracks, are entirely downloaded in advance andstored locally on the user device before playback can commence. In someembodiments, the soundtrack and audio track files are combined with theelectronic text file as a single multimedia data file for download andplayback. The user playback system processes the locally storeddownloaded file or files and co-ordinates playback of the audio inaccordance with a reading position counter that estimates the user'sreading position in the electronic text. This user playback systemconfiguration utilizes heavy processing and memory resources of thehardware device to deliver the synchronised soundtrack. In particular,the soundtrack data file and audio track files may be very large in sizeand can occupy a significant amount of the local storage on the hardwaredevice, and this limits the user's ability to store a library ofsoundtracks for their e-books. The downloading of these large files ontodevices, especially mobile devices, can also be costly if beingdownloaded over a user's mobile or other subscribed data connection.Additionally the user playback system uses processing resources to bothdetermine the current reading position and then retrieve and coordinateplayback of the audio track file in a synchronised manner.

In this specification where reference has been made to patentspecifications, other external documents, or other sources ofinformation, this is generally for the purpose of providing a contextfor discussing the features of the invention. Unless specifically statedotherwise, reference to such external documents is not to be construedas an admission that such documents, or such sources of information, inany jurisdiction, are prior art, or form part of the common generalknowledge in the art.

It is an object of the invention to provide an improved system for thedelivery of synchronised soundtracks for electronic media content, or atleast provide the public with a useful choice.

SUMMARY OF THE INVENTION

In a first aspect, the present invention broadly consists in a method ofstreaming a compiled soundtrack to a remote user device for a reader ofelectronic media content, the soundtrack defined by multiple audioregions, each audio region defined by: an audio track for playback inthe audio region, a start position in the electronic media contentcorresponding to where the playback of the audio region is to begin, anda stop position in the electronic media content corresponding to wherethe playback of the audio region is to cease, the method comprising:

-   -   receiving from a remote user device over a network control data        indicative of the user's reading speed; and    -   streaming a compiled soundtrack to the user device over the        network based on the received control data.

In a first form, the compiled soundtrack is a pre-compiled soundtrackand is selected from a range of pre-compiled soundtracks that have beenpre-compiled at different reading speeds. Preferably, the method furthercomprises selecting the pre-compiled soundtrack for streaming having anassociated reading speed that most closely matches the control dataindicative of the user's reading speed.

In a second form, the compiled soundtrack is a real-time compiledsoundtrack and the method further comprises compiling the soundtrack inreal-time based on the control data indicative of the user's readingspeed. Preferably, the soundtrack is compiled progressively and themethod comprises streaming the compiled portion of the soundtrack whilethe remaining portion is being compiled.

Preferably, the method further comprises receiving control data from theremote user device indicative of the user's desired commencement readingposition and commencing streaming of the compiled soundtrack to the userdevice at a data position within the compiled soundtrack thatcorresponds to the user's desired commencement reading position. By wayof example, the data position may be at the start of the compiledsoundtrack or at an offset position within the compiled soundtrack.

Preferably, the method further comprises receiving control data from theremote user device indicative of a change in reading position of theuser to a new reading position in the electronic media content, andmodifying the streaming of the compiled soundtrack based on the receivedcontrol data. More preferably, modifying the streaming of the compiledsoundtrack comprises adjusting the streaming of the compiled soundtrackto a new data position within the compiled soundtrack that correspondsto the new reading position.

Preferably, the method further comprises receiving control data from theremote user device indicative of a change in reading speed of the userto a new user reading speed, and modifying the streaming of the compiledsoundtrack based on the received control data.

In the first form, in which the compiled soundtrack is a pre-compiledsoundtrack, modifying the streaming of the compiled soundtrack based ona new user reading speed comprises selecting a new pre-compiledsoundtrack having a reading speed that more closely matches the new userreading speed and commencing streaming of the new pre-compiledsoundtrack. Preferably, the method further comprises implementing orinitiating a transition protocol for transitioning the streaming fromthe current pre-compiled soundtrack to the new selected pre-compiledsoundtrack. In one example, the transition protocol comprises receivingcontrol data from the remote user device indicative of the user'scurrent playback position in the streamed soundtrack and commencingstreaming of the new selected pre-compiled soundtrack at a data positioncorresponding to the user's current playback position. In anotherexample, the transition protocol comprises analyzing the remainingunstreamed portion of the current pre-compiled soundtrack andidentifying a transition position based on transition criteria, andcommencing streaming of the new selected pre-compiled soundtrack at adata position corresponding to the identified transition position.

In the second form, in which the compiled soundtrack is a real-timecompiled soundtrack, modifying the streaming of the compiled soundtrackbased on a new user reading position comprises commencing compiling ofthe soundtrack at the new user reading speed and commencing streaming ofthe new compiled soundtrack. Preferably, the method further comprisesimplementing or initiating a transition protocol for transitioning thestreaming from the current real-time compiled soundtrack to the newreal-time compiled soundtrack. In one example, the transition protocolcomprises receiving control data from the remote user device indicativeof the user's current playback position in the streamed soundtrack andcommencing streaming of the new real-time compiled soundtrack at a dataposition corresponding to the user's current playback position. Inanother example, the transition protocol comprises analyzing theremaining unstreamed portion of the current real-time compiledsoundtrack and identifying a transition position based on transitioncriteria, and commencing streaming of the new real-time compiledsoundtrack at a data position corresponding to the identified transitionposition.

In some embodiments, the method comprises streaming the compiledsoundtrack as a single digital audio file. In other embodiments, themethod comprises streaming the compiled soundtrack as a plurality ofdigital audio files, for concurrent playback at the remote user device.For example, the audio regions of the soundtrack may be categorised intodifferent types, such as music, ambience, or effects, and the compiledsoundtrack may be provided as separate audio files, each audio filecomprising the audio regions associated with one type.

In a second aspect, the invention broadly consists in a soundtrackserver for streaming a compiled soundtrack to a remote user device for areader of electronic media content, the soundtrack defined by multipleaudio regions, each audio region defined by: an audio track for playbackin the audio region, a start position in the electronic media contentcorresponding to where the playback of the audio region is to begin, anda stop position in the electronic media content corresponding to wherethe playback of the audio region is to cease, the soundtrack serverbeing configured to:

-   -   receive from a remote user device over a network control data        indicative of the user's reading speed; and    -   stream a compiled soundtrack to the user device over the network        based on the received control data.

The soundtrack server may further comprise any one or more of thefeatures mentioned in respect of the first aspect of the invention.

In a third aspect, the present invention broadly consists in a method ofplaying a compiled soundtrack on a user device for a reader ofelectronic media content, the soundtrack defined by multiple audioregions, each audio region defined by: an audio track for playback inthe audio region, a start position in the electronic media contentcorresponding to where the playback of the audio region is to begin, anda stop position in the electronic media content corresponding to wherethe playback of the audio region is to cease, the method comprising:

-   -   sending control data to a remote server over a network        indicative of the user's reading speed;    -   receiving an incoming stream of a compiled soundtrack from the        remote server over the network, the compiled soundtrack having        an associated reading speed that corresponds to the sent control        data;    -   processing the streamed compiled soundtrack and commencing        playback of the soundtrack on the user device.

Preferably, the method further comprises sending control data to theremote server indicative of the user's desired commencement readingposition such that the received stream of the compiled soundtrackcommences at a data position within the compiled soundtrackcorresponding to the user's desired commencement reading position.

Preferably, the method further comprises buffering the incoming streamof the compiled soundtrack into memory on the user device ready forplayback. More preferably, the method comprises receiving user inputindicative of a new reading position with the electronic media content,determining whether that new reading position corresponds to a playbackposition that is within the buffered portion of the compiled soundtrack,and either transitioning playback to the new playback position in thebuffered compiled soundtrack if it is within the buffer or sendingcontrol data indicative of the new reading position to the remote serverto initiate an updated stream of the compiled soundtrack at the newreading position if it is outside the buffer. The method may furthercomprise implementing a transition protocol to transition the playbackto the new playback position with the buffer or to the updated incomingstream.

Preferably, the method further comprises receiving user input indicativeof a change in reading speed or detecting a change in reading speed, andsending control data indicative of the updated reading speed to theremote server to initiate an updated stream of the compiled soundtrackat the new reading speed. The method may further comprise implementing atransition protocol to transition playback to the updated incomingstream.

In a fourth aspect, the present invention broadly consists in a userdevice for playing a compiled soundtrack for a reader of electronicmedia content, the soundtrack defined by multiple audio regions, eachaudio region defined by: an audio track for playback in the audioregion, a start position in the electronic media content correspondingto where the playback of the audio region is to begin, and a stopposition in the electronic media content corresponding to where theplayback of the audio region is to cease, the user device comprising:

-   -   a user interface configured to receive user input to control the        device;    -   an audio output system that is operable to generate audio output        for the user to hear; and    -   a processor configured to:        -   send control data to a remote server over a network            indicative of the user's reading speed;        -   receive an incoming stream of a compiled soundtrack from the            remote server over the network, the compiled soundtrack            having an associated reading speed that corresponds to the            sent control data; and        -   process the streamed compiled soundtrack to commence            playback of the soundtrack over the audio output system in            response to user input.

In one form, the device further comprises a display configured todisplay the electronic media content, such that both the audiblesoundtrack and visible electronic media content are provided to the useron the same user device.

In another form, the user device is configured to provide the audiblesoundtrack to the user and is in data communication with a secondseparate user device that comprises a display and which is configured toprovide the user with the visible electronic media content. By way ofexample, the data communication between the pair of user devices may be,direct or indirect, including via Radio Frequency (RF), Near-fieldcommunication (NFC), Bluetooth, WiFi or other such wirelesscommunication mediums or protocols, or alternatively a wiredcommunication medium may be provided if the pair of user devices areconnected via a cable or cables.

The user device(s) may further comprise any one or more of the featuresmentioned in respect of the third aspect of the invention.

In a fifth aspect, the invention broadly consists in a system forstreaming a compiled soundtrack to a remote user device for a reader ofelectronic media content, the soundtrack defined by multiple audioregions, each audio region defined by: an audio track for playback inthe audio region, a start position in the electronic media contentcorresponding to where the playback of the audio region is to begin, anda stop position in the electronic media content corresponding to wherethe playback of the audio region is to cease, the system comprising:

-   -   a user device comprising:        -   a user interface configured to receive user input to control            the device;        -   an audio output system that is operable to generate audio            output for the user to hear;        -   a data communication module that is configured to            communicate with a remote server via a network; and        -   a processor configured to:            -   send control data to the remote server indicative of the                user's reading speed;            -   receive an incoming stream of a compiled soundtrack from                the remote server over the network, the compiled                soundtrack having an associated reading speed that                corresponds to the sent control data; and            -   process the streamed compiled soundtrack to commence                playback of the soundtrack over the audio output system                in response to user input; and        -   a remote server which is configured to:            -   receive from the user device the control data indicative                of the user's reading speed; and            -   stream a compiled soundtrack to the user device based on                the received control data.

In a first embodiment, the user device further comprises a displayconfigured to display the electronic media content. In this embodiment,the user device may be configured to run a first application whichcontrols the retrieval, processing and display of the electronic mediacontent from data files stored locally or on an accessible database orretrieved or streamed from online or the internet or an electronic mediacontent server (such as an electronic e-book server, store or library),and a second application that is configured to receive and process thestreamed compiled soundtrack from the remote server and control playbackof the compiled soundtrack over the audio output system of the userdevice. By way of example, the first application may be an Internetbrowser viewing a webpage, an e-reader software for viewing an e-book,or any other text, document or electronic media content viewer. By wayof example, the second application may be a customised soundtrack playeror an Internet browser accessing a soundtrack player website hosted bythe soundtrack server

In a first form, the first and second applications may operateindependently without the exchange of data. In a second form, the firstapplication may send data to or make data available for retrieval by thesecond application, the data being indicative of user readinginformation including, for example, user's current reading position,reading speed, data indicative of page turns, scrolling information,arbitrary reading position updates, and/or data indicative of the titleor details identifying the item of electronic media content being read.The data communication link between the first and second applicationsmay be via an application programming interface (API) or the like.

In this embodiment, the first application running on the user device isconfigured to display the electronic media content in response to userinteraction, and the second application is configured to control theplayback of the compiled soundtrack for the electronic media content viauser interaction. By way of example, the user device is configured to:

-   -   load the electronic media content for display via the first        application;    -   send control data to the remote server to request a streamed        compiled soundtrack that corresponds to the electronic media        content via the second application; and    -   receive and playback the stream compiled soundtrack received        from the remote server via the second application.

In a second embodiment, the user device further comprises a displayconfigured to display the electronic media content. In this embodiment,the user device is configured to run a single multimedia applicationwhich is configured to control the display of the electronic mediacontent and playback of the streamed compiled soundtrack via userinteraction.

In a first form, the multimedia application may be an Internet browserthat is configured to receive and process the electronic media contentand streamed compiled soundtrack received from a remote server in theform of a web site server. In one form, the Internet browser may receivethe electronic media content and streamed compiled soundtrack in asingle combined data stream or alternatively in another form theelectronic media content and streamed compiled soundtrack may be sent inseparate data streams. In one form, the electronic media content may bedownloaded entirely for display or alternatively downloaded anddisplayed in chunks, or streamed. The soundtrack streaming and/orplayback functions may be performed by the Internet browser itself or aplug-in application program to the browser, or an embedded applicationin the web page.

In a second form, the multimedia application may be a custom applicationthat is configured to retrieve and display electronic media content andcontrol playback of the streamed compiled soundtrack from a multimediacontent server. In one form, the multimedia content server may provideelectronic media content in the form of an e-book and the streamedcompiled soundtrack corresponds to the soundtrack for the e-book. In afirst form, the e-book data may be sent in a separate data stream,whether downloaded in advance or in chunks, or streamed (and the audiostreams may be sent in a separate data stream). In another form, thee-book data and audio stream may be sent in a combined multimedia datastream.

In a third embodiment, the user device is configured to provide theaudible soundtrack to the user and is in data communication with asecond separate user device that comprises a display and which isconfigured to provide the user with the visible electronic mediacontent. By way of example, the data communication between the pair ofuser devices may be, direct or indirect, including via Radio Frequency(RF), Near-field communication (NFC), Bluetooth, WiFi or other suchwireless communication mediums or protocols, or alternatively a wiredcommunication medium may be provided if the pair of user devices areconnected via a cable or cables.

In this embodiment, the second separate user device (text displaydevice) is configured to send data to or make data available forretrieval by the first user device (audio playback device), the databeing indicative of user reading information including, for example,user's current reading position, reading speed, data indicative of pageturns, scrolling information, arbitrary reading position updates, and/ordata indicative of the title or details identifying the item ofelectronic media content being read.

The system may have any one or more of the features mentioned in respectof the first-fourth aspects of the invention.

In a sixth aspect, the present invention broadly consists in a method ofstreaming an uncompiled soundtrack to a remote user device for a readerof electronic media content, the soundtrack defined by multiple audioregions, each audio region defined by: an audio track for playback inthe audio region, a start position in the electronic media contentcorresponding to where the playback of the audio region is to begin, anda stop position in the electronic media content corresponding to wherethe playback of the audio region is to cease, the method comprising:

-   -   sending a soundtrack data file to the remote user device over a        network, the soundtrack data file comprising data defining the        audio regions of the soundtrack;    -   receiving control data from the remote user device over the        network requesting one or more audio track files corresponding        to the audio regions of the soundtrack; and    -   sending one or more audio track files to the remote user device        over the network in response to the received control data.

In one form, the method comprises receiving arbitrary or periodiccontrol data requesting individual audio track files, and sending therequested audio track files one by one to the remote user device uponrequest.

In another form, the method comprises receiving control data with asingle request for all audio track files associated with the soundtrack,and sending all audio track files to the remote user device, either oneby one, in batches or otherwise. Preferably, the method comprisessending the audio track files in order of their playback within thesoundtrack.

In another form, the method comprises receiving arbitrary or periodiccontrol data comprising requests for batches of audio track files, andsending the requested batches of audio track files to the remote userdevice upon request.

In a seventh aspect, the invention broadly consists in a soundtrackserver for streaming an uncompiled soundtrack to a remote user devicefor a reader of electronic media content, the soundtrack defined bymultiple audio regions, each audio region defined by: an audio track forplayback in the audio region, a start position in the electronic mediacontent corresponding to where the playback of the audio region is tobegin, and a stop position in the electronic media content correspondingto where the playback of the audio region is to cease, the soundtrackserver being configured to:

-   -   send a soundtrack data file to the remote user device over a        network, the soundtrack data file comprising data defining the        audio regions of the soundtrack;    -   receive control data from the remote user device over the        network requesting one or more audio track files corresponding        to the audio regions of the soundtrack; and    -   send one or more audio track files to the remote user device        over the network in response to the received control data.

The soundtrack server may further comprise any one or more of thefeatures mentioned in respect of the sixth aspect of the invention.

In a eighth aspect, the present invention broadly consists in a methodof playing an uncompiled soundtrack on a user device for a reader ofelectronic media content, the soundtrack defined by multiple audioregions, each audio region defined by: an audio track for playback inthe audio region, a start position in the electronic media contentcorresponding to where the playback of the audio region is to begin, anda stop position in the electronic media content corresponding to wherethe playback of the audio region is to cease, the method comprising:

-   -   sending control data to a remote server over a network        requesting a soundtrack for playback;    -   receiving a soundtrack data file from the remote serve in        response to the sent control data, the soundtrack data file        defining the audio regions of the soundtrack;    -   processing the soundtrack data file to determine at least the        next audio region due for playback based on a user reading        position variable indicative of the user's reading position in        the electronic media content;    -   sending periodic or arbitrary control data to one or more audio        track file sources requesting at least the audio track file        corresponding to the next audio region due for playback;    -   receiving and storing requested audio track files from the one        or more audio track file sources in memory; and    -   co-ordinating playback of the stored audio track files as        defined by their associated audio regions in the soundtrack data        file based on the user reading position variable.

In an embodiment, the soundtrack data file comprises source dataindicative of the one or more audio track file sources to request eachaudio track file from. In some forms, the soundtrack data file maydefine a single audio track file source from which to request all theaudio track files. In other forms, the soundtrack data file may definetwo or more different audio track file sources from which to request theaudio track files.

In a first embodiment, the audio track file source is the remote server,and the method comprises requesting all audio track files from theremote server.

In a second embodiment, the audio track file sources comprise the remoteserver and one or more other audio track file sources, and the methodcomprises requesting the audio track files from the remote server or theone or more other audio track file sources based on the source data inthe soundtrack data file. For example, some audio track files may besourced from the remote server, while other audio track files may besourced from one or more other audio track file sources.

In a third embodiment, the one or more audio track file sources areseparate to the remote server, and the method comprises requesting allaudio track files from the one or more separate audio track filesources. For example, all audio track files are sourced from somewhereother than the remote server.

In one form, the method comprises sending control data to the one ormore audio track file sources requesting individual audio track filesone by one.

In another form, the method comprises sending control data to the one ormore audio track file sources comprising a single request for all audiotrack files associated with the audio regions of the soundtrack.

In another form, the method comprises sending control data to the one ormore audio track file sources requesting batches of audio track files.

In another form, the method comprises maintaining a sliding downloadwindow relative to the user reading position variable (which correspondsto the playback position within the soundtrack), and sending controldata to the one or more audio track file sources requesting all audiotrack files associated with audio regions falling within the slidingdownload window such that the all audio track files associated withaudio regions in the sliding window are received and stored ready forfuture playback based on the user reading position variable. The size ofthe sliding download window may be configurable.

In one form, the method further comprises deleting from memory the audiotrack files after their corresponding audio regions have completedplayback either immediately or after a configurable predetermined delay.

In one form, the method further comprises analyzing the soundtrack datafile for audio track files that are repeated in more than one audioregion of the soundtrack, and tagging those audio track files asrepeated audio tracks upon receipt and storage. More preferably, themethod comprises retaining each repeated audio tracks in memory untilthe last audio region associated with the audio track has completedplayback.

In a ninth aspect, the present invention broadly consists in a userdevice for playing an uncompiled soundtrack for a reader of electronicmedia content, the soundtrack defined by multiple audio regions, eachaudio region defined by: an audio track for playback in the audioregion, a start position in the electronic media content correspondingto where the playback of the audio region is to begin, and a stopposition in the electronic media content corresponding to where theplayback of the audio region is to cease, the user device comprising:

-   -   a user interface configured to receive user input to control the        device;    -   an audio output system that is operable to generate audio output        for the user to hear; and    -   a processor configured to:        -   send control data to a remote server over a network            requesting a soundtrack for playback;        -   receive a soundtrack data file from the remote serve in            response to the sent control data, the soundtrack data file            defining the audio regions of the soundtrack;        -   process the soundtrack data file to determine at least the            next audio region due for playback based on a user reading            position variable indicative of the user's reading position            in the electronic media content;        -   send periodic or arbitrary control data to one or more audio            track file sources requesting at least the audio track file            corresponding to the next audio region due for playback;        -   receive and store requested audio track files from the one            or more audio track file sources in memory on the user            device; and        -   co-ordinate playback of the stored audio track files as            defined by their associated audio regions in the soundtrack            data file based on the user reading position variable.

In one form, the device further comprises a display configured todisplay the electronic media content, such that both the audiblesoundtrack and visible electronic media content are provided to the useron the same user device.

In another form, the user device is configured to provide the audiblesoundtrack to the user and is in data communication with a secondseparate user device that comprises a display and which is configured toprovide the user with the visible electronic media content. By way ofexample, the data communication between the pair of user devices may be,direct or indirect, including via Radio Frequency (RF), Near-fieldcommunication (NFC), Bluetooth, WiFi or other such wirelesscommunication mediums or protocols, or alternatively a wiredcommunication medium may be provided if the pair of user devices areconnected via a cable or cables.

The user device(s) may further comprise any one or more of the featuresmentioned in respect of the eighth aspect of the invention.

In a tenth aspect, the invention broadly consists in a system forstreaming an uncompiled soundtrack to a remote user device for a readerof electronic media content, the soundtrack defined by multiple audioregions, each audio region defined by: an audio track for playback inthe audio region, a start position in the electronic media contentcorresponding to where the playback of the audio region is to begin, anda stop position in the electronic media content corresponding to wherethe playback of the audio region is to cease, the system comprising:

-   -   a user device comprising:        -   a user interface configured to receive user input to control            the device;        -   an audio output system that is operable to generate audio            output for the user to hear;        -   a data communication module that is configured to            communicate with a remote server via a network; and        -   a processor configured to:            -   send control data to a remote server over a network                requesting a soundtrack for playback;            -   receive a soundtrack data file from the remote serve in                response to the sent control data, the soundtrack data                file defining the audio regions of the soundtrack;            -   process the soundtrack data file to determine at least                the next audio region due for playback based on a user                reading position variable indicative of the user's                reading position in the electronic media content;            -   send periodic or arbitrary control data to the remote                server requesting at least the audio track file                corresponding to the next audio region due for playback;            -   receive and store requested audio track files from the                remote server in memory on the user device; and            -   co-ordinate playback of the stored audio track files as                defined by their associated audio regions in the                soundtrack data file based on the user reading position                variable; and    -   a remote server which is configured to:        -   send the soundtrack data file to the user device over a            network in response to control data from the user device;        -   receive control data from the user device over the network            requesting one or more audio track files corresponding to            the audio regions of the soundtrack; and        -   send one or more audio track files to the user device over            the network in response to the received control data.

In a first embodiment, the user device further comprises a displayconfigured to display the electronic media content. In this embodiment,the user device may be configured to run a first application whichcontrols the retrieval, processing and display of the electronic mediacontent from data files stored locally or on an accessible database orretrieved or streamed from online or the internet or an electronic mediacontent server (such as an electronic e-book server, store or library),and a second application that is configured to receive and process thestreamed uncompiled soundtrack from the remote server and controlplayback of the compiled soundtrack over the audio output system of theuser device. By way of example, the first application may be an Internetbrowser viewing a webpage, an e-reader software for viewing an e-book,or any other text, document or electronic media content viewer. By wayof example, the second application may be a customised soundtrack playeror an Internet browser accessing a soundtrack player website hosted bythe soundtrack server.

In a first form, the first and second applications may operateindependently without the exchange of data. In a second form, the firstapplication may send data to or make data available for retrievable bythe second application, the data being indicative of user readinginformation including, for example, user's current reading position,reading speed, data indicative of page turns, scrolling information,arbitrary reading position updates, and/or data indicative of the titleor details identifying the item of electronic media content being read.The data communication link between the first and second applicationsmay be via an application programming interface (API) or the like.

In this embodiment, the first application running on the user device isconfigured to display the electronic media content in response to userinteraction, and the second application is configured to control theplayback of the uncompiled soundtrack for the electronic media contentvia user interaction. By way of example, the user device is configuredto:

-   -   load the electronic media content for display via the first        application;    -   send control data to the remote server to request a streamed        uncompiled soundtrack that corresponds to the electronic media        content via the second application; and    -   receive and playback the streamed uncompiled soundtrack received        from the remote server via the second application.

In a second embodiment, the user device further comprises a displayconfigured to display the electronic media content. In this embodiment,the user device is configured to run a single multimedia applicationwhich is configured to control the display of the electronic mediacontent and playback of the streamed uncompiled soundtrack via userinteraction.

In a first form, the multimedia application may be an Internet browserthat is configured to receive and process the electronic media contentand streamed compiled soundtrack received from a remote server in theform of a web site server. In one form, the Internet browser may receivethe electronic media content and streamed uncompiled soundtrack in asingle combined data stream or alternatively in another form theelectronic media content and streamed uncompiled soundtrack may be sentin separate data streams. In one form, the electronic media content maybe downloaded entirely for display or alternatively downloaded anddisplayed in chunks, or streamed. The soundtrack streaming and/orplayback functions may be performed by the Internet browser itself or aplug-in application program to the browser, or an embedded applicationin the web page.

In a second form, the multimedia application may be a custom applicationthat is configured to retrieve and display electronic media content andcontrol playback of the streamed compiled soundtrack from a multimediacontent server. In one form, the multimedia content server may provideelectronic media content in the form of an e-book and the streameduncompiled soundtrack corresponds to the soundtrack for the e-book. In afirst form, the e-book data may be sent in a separate data stream,whether downloaded in advance or in chunks, or streamed (and the audiostreams may be sent in a separate data stream). In another form, thee-book data and audio stream may be sent in a combined multimedia datastream.

In a third embodiment, the user device is configured to provide theaudible soundtrack to the user and is in data communication with asecond separate user device that comprises a display and which isconfigured to provide the user with the visible electronic mediacontent. By way of example, the data communication between the pair ofuser devices may be, direct or indirect, including via Radio Frequency(RF), Near-field communication (NFC), Bluetooth, WiFi or other suchwireless communication mediums or protocols, or alternatively a wiredcommunication medium may be provided if the pair of user devices areconnected via a cable or cables.

In this embodiment, the second separate user device (text displaydevice) is configured to send data to or make data available forretrieval by the first user device (audio playback device), the databeing indicative of user reading information including, for example,user's current reading position, reading speed, data indicative of pageturns, scrolling information, arbitrary reading position updates, and/ordata indicative of the title or details identifying the item ofelectronic media content being read.

The system may have any one or more of the features mentioned in respectof the fifth-ninth aspects of the invention.

In another aspect, the invention broadly consists in a computer-readablemedium having stored thereon computer readable instructions that, whenexecuted on a processing device, cause the processing device to performany one or more of the methods defined in the previous aspects of theinvention.

Definitions

The term “soundtrack” as used in this specification and claims isintended to mean, unless the context suggests otherwise, a soundtrackfor playback while reading electronic media content and of the typedescribed in WO2013/015694 which comprises multiple audio regions thatare configured for playback at predetermined reading positions orregions within the electronic media content, and where the playbacktiming of the soundtrack is synchronized with the user based on theirestimated reading position and/or reading speed.

The phrases “streaming”, “stream”, or “streamed” as used in thisspecification and claims is intended to mean, unless the contextsuggests otherwise, the streaming of the soundtrack to a user devicefrom a remote server in any form which enables the user to commenceplayback of the soundtrack without downloading the entire soundtrack,whether the soundtrack is in the form of a compiled soundtrackready-to-play or an uncompiled soundtrack, and includes any suitableform of streaming the data such as continuous streaming of thesoundtrack, sending the soundtrack in batches, chunks, portions orsections, and whether the streaming is controlled by the server, userdevice, or a combination of these.

The phrase “compiled soundtrack” as used in this specification andclaims is intended to mean, unless the context suggests otherwise, asoundtrack compiled in the form of a ready-to-play digital audio file,the soundtrack being compiled at a configurable reading speed to modifythe duration of the soundtrack to suit readers having different readingspeeds, the compiled soundtrack may be compiled, encoded and/orcompressed into any digital audio format for playback by a digital audioplayer or audio player software, and including real-time compiled audiofiles or pre-compiled audio files.

The phrase “uncompiled soundtrack” as used in this specification andclaims, is intended to mean, unless the context suggests otherwise, asoundtrack in the form of a package of files comprising a soundtrackdata file defining the playback data for the multiple audio regions ofthe soundtrack and the audio track files associated with each audioregion, the uncompiled soundtrack being configured for playback by asoundtrack player that processes the soundtrack data file andco-ordinates synchronized playback of the audio track files of the audioregions with the user's reading based on data, a counter, or a variableindicative of the user's reading position within the electronic mediacontent.

The phrase “user device” as used in this specification and claims isintended to mean, unless the context suggests otherwise, any form ofprogrammable electronic hardware device, platform or system including,but not limited to, portable or non-portable consumer electronic devicessuch as smartphones, cellphones, tablets, e-Reader or e-book devices,laptops, and notebooks, gaming machines or consoles, smart televisions,desktop computers or the like.

The term “network” as used in this specification and claims is intendedto mean, unless the context suggests otherwise, any data network or datacommunication channel or combination of networks or channels over whicha user device may send and/or receive data to another device, whether aremote server or servers or other user devices, including, but notlimited to, the internet, an intranet or extranet, a Local Area Network(LAN), and/or a Wireless Local Area Network (WLAN), whether accessedover a wired or wireless medium or a combination of these.

The term “server” as used in this specification and claims is intendedto mean, unless the context suggests otherwise, any form of electronicserver that is capable or configured to serve or stream soundtracks to auser device over a network, whether a website server, dedicatedsoundtrack server, electronic media content server, or any other form ofserver, whether a dedicated server or other hardware system or platformperforming a server function, and the server may be configured toservice or host one or more client or user devices, and including anyform of data communication between the user device and server, whetherwired, wireless or a combination of these.

The phrase “electronic media content” as used in this specification andclaims is intended to mean, unless the context suggests otherwise, anyelectronic media content comprising electronic text, whether alone or incombination with static or animated imagery or graphics, the electronicmedia content being in any format or file-type capable of being renderedon an electronic display, and including, but not limited to, anelectronic book or e-book, a webpage, digital video, or slide showpresentation.

The term “comprising” as used in this specification and claims means“consisting at least in part of”. When interpreting each statement inthis specification and claims that includes the term “comprising”,features other than that or those prefaced by the term may also bepresent. Related terms such as “comprise” and “comprises” are to beinterpreted in the same manner.

As used herein the term “and/or” means “and” or “or”, or both.

As used herein “(s)” following a noun means the plural and/or singularforms of the noun.

The invention consists in the foregoing and also envisages constructionsof which the following gives examples only.

In the following description, specific details are given to provide athorough understanding of the embodiments. However, it will beunderstood by one of ordinary skill in the art that the embodiments maybe practiced without these specific details. For example, softwaremodules, functions, circuits, etc., may be shown in block diagrams inorder not to obscure the embodiments in unnecessary detail. In otherinstances, well-known modules, structures and techniques may not beshown in detail in order not to obscure the embodiments.

Also, it is noted that the embodiments may be described as a processthat is depicted as a flowchart, a flow diagram, a structure diagram, ora block diagram. Although a flowchart may describe the operations as asequential process, many of the operations can be performed in parallelor concurrently. In addition, the order of the operations may berearranged. A process is terminated when its operations are completed. Aprocess may correspond to a method, a function, a procedure, asubroutine, a subprogram, etc., in a computer program. When a processcorresponds to a function, its termination corresponds to a return ofthe function to the calling function or a main function.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will be described by way of example onlyand with reference to the drawings, in which:

FIG. 1 is a schematic data illustration of a portion of text and theassociated audio regions of a soundtrack configured for playback duringthe reading of the electronic text;

FIG. 2 is a schematic data illustration similar to FIG. 1 except showinga soundtrack of the invention configured for electronic media contentcomprising text and images;

FIG. 3A is a schematic illustration of a soundtrack delivery system inaccordance with a first embodiment of the invention in which separateapplications on the user device control display of the electronic mediacontent and playback of the soundtrack based on an audio stream from aremote server;

FIG. 3B shows a schematic illustration of a soundtrack delivery systemin accordance with a second embodiment of the invention, the systememploying a single multimedia application to control the display of theelectronic media content and playback of the soundtrack based onelectronic media content and audio data stream from a remote multimediacontent server;

FIG. 4 shows a schematic illustration of a soundtrack delivery system inaccordance with a third embodiment of the invention, the systememploying a pair of user devices in data communication, one user devicecontrolling the display of the electronic media content and the otheruser device controlling playback of the soundtrack based on an audiodata stream from a remote server;

FIG. 5A is a schematic illustration of a soundtrack delivery system inaccordance with a first form of the invention, the system delivering aselected pre-compiled soundtrack audio file over an audio stream from asoundtrack server to a remote user device, the soundtrack being selectedfrom a library of soundtracks pre-compiled at a range of differentreading speeds;

FIG. 5B is a flow diagram showing the re-synchronisation to a newreading position in the system of FIG. 5A;

FIG. 5C is a flow diagram showing the transition protocols for a changein reading speed in the system of FIG. 5A;

FIG. 6 is a schematic illustration of a soundtrack delivery system inaccordance with a second form of the invention, the system delivering areal-time compiled soundtrack audio file over an audio stream from asoundtrack server to a remote device; and

FIG. 7 is a schematic illustration of a soundtrack delivery system inaccordance with a third form of the invention, the system delivering anuncompiled soundtrack over an audio stream from a soundtrack server to aremote user device.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 1. Overview ofSynchronised Soundtracks

This invention relates to delivery of soundtrack of the type describedin WO2013/015694, as referred to above. The soundtracks are created toenhance the user's reading experience for an item of electronic text.

The soundtracks may be applied to electronic text in any format orfile-type, including but not limited to those associated with thefollowing file extensions: aeh, djvu, epub, fb2, html, xhtml, azw, lit,prc, mobi, exe, pdb, txt, pdb, pdf, ps, rgo, pdb, tr2, tr3, wol, rtf,doc, docx, asc, lit, wri, odt, text, pwi, rtx, sdw, strings, utf8,utf16, wp*, wpw, wri, wtx, asc, csv, lwp, dtp, indd, pmd, ppp, pub, fm,and any type of mark up language types, including but not limited toSGML, HTML, XHTML, HTML5, XML, and LaTex. The electronic text may be inthe form of any electronic media content whether comprising text only ortext in combination with imagery or graphics, whether static or moving(e.g. video or other animations). By way of example, but not limited to,the electronic media content may be in the form of an e-book, a webpage,comic or any other electronic media content format.

The soundtrack is customised to the electronic media content and inparticular customized to the content of the electronic text to enhancethe reading experience for a reader of the electronic text. Thesoundtrack may comprise audio tracks of different types and may beconfigured to play concurrently, sequentially, or both while the user isreading the electronic text. Playback of the soundtrack is synchronizedwith the reading speed (e.g. words per minute or scrolling speed or someother metric) of the user. In one example, the electronic text may be afiction or non-fiction book, and the soundtrack may comprise acombination of different audio types, including ambient noise, soundeffects, and background music that may be configured to play back at aspecific point or over specific portions of the electronic text in asynchronized manner as the user is reading those portions of theelectronic text.

The soundtrack comprises multiple audio regions that are designated toplayback during designated text portions of the electronic text. Eachaudio region comprises audio data defining the audio track to playbackand audio properties defining the desired nature of the playback. Theaudio data specifies a start text position in the electronic text atwhich the audio region is to commence playback and a stop text positionat which the audio region is to cease playback. In one embodiment, theelectronic text is designated a numerical text index based on the numberor words or characters in the electronic text, and the start and stoptext positions are defined by corresponding start and stop index valuesrespectively. In one example, the indexing of the text is based on thenumber of words in the electronic text such that each word has a uniqueindex value representing its position in the electronic text. The indexvalues for each word may correspond to their word number or word countin the electronic text. In this example, the audio regions playbackbetween start and stop index values representing the selected start andstop words in the electronic text for that audio region.

When the user is reading the electronic text, a reading position counteror identifier is maintained which corresponds to text position in theelectronic text at which the user is estimated as currently reading. Thereading position counter increments or changes at a rate at leastpartially based on the user's reading speed. This reading positioncounter allows the playback of the audio regions of the soundtrack to besynchronized with the user's reading speed such that playback of anaudio region is triggered when the reading position counter matches thestart index value of the start word for the audio region, and likewiseceases when the reading position counter matches the stop index value ofthe stop word of the audio region.

In alternative embodiments, other reading position referencing schememay be employed in contrast to word or character based indexing. Inother embodiments, the start and stop positions of each audio region maybe defined by any other document or text position referencing scheme orsystem. One such example is the ePub canonical fragment identifier(CFI). In such an example, the start and stop positions in the audioregion will be defined by the CFI parameter or ranges, and a readingposition variable would be maintained in the CFI format which changes orincrement based on the user's reading speed to trigger the playback ofthe audio regions of the soundtrack when their CFI value match thereading position variable.

The soundtrack comprises multiple audio regions and playback of two ormore regions may partially or entirely overlap such that two or moreaudio regions can playback concurrently if desired. The soundtrack isprovided with multiple channels for playback and overlapping audioregions are designated different channels in the set of availablechannels to allow for concurrent playback. This enables different audiotracks, such as sound effects, ambient noise, or background music to beconcurrently played and mixed together to provide an overall soundtrackthat enhances the user's reading experience for that particularelectronic text. Typically, the audio data for each audio regionincludes an audio type, such as “music”, “ambience”, or “effect”. Theselected audio type is intended to define the nature of the audioregion. For example, if an audio region is for background music, thenthe audio type selected will be “music”. If the audio region is a soundeffect, then the “effect” audio type is selected. If the audio region isintended to be an ambient sound or noise, then the “ambience” audio typeis selected.

By way of example, FIG. 1 shows a portion of electronic media contentcomprising electronic text only. FIG. 1 also overlays the text with aschematic representation of a number of audio regions 10 configured forplayback during reading of the electronic text in the schematicrepresentation, the audio regions 10 are each represented by a bar orunderline that extends below the portion of text to which it relates.The start 10 a of each bar or underline of the audio region representsthe start position at which the audio track of that audio region willcommence playback and the end 10 b represents the position in theelectronic text at which the audio track of the audio region will ceaseplayback. Each audio region may extend over a single word or group ofwords, a line, multiple lines or even pages. As shown the audio regionsmay overlap such that they playback concurrently.

Referring to FIG. 2, the electronic media content may comprisemultimedia content comprising text and imagery, whether static ormoving. FIG. 2 shows such an electronic text containing static images20. For such electronic media content, the soundtrack audio region mayhave start and stop positions which either correspond to text orimagery. For example an audio region may be configured to start on aparticular word, and then end when a user is expected to be looking at aparticular image embedded within the text. Alternatively, an audioregion may be configured to commence playback when a user is expected tobe viewing a particular embedded image. It will be appreciated that forimage-based multi-media content such as comics, the reading positionidentifier may identify the image the user is expected to be viewing,and the reading speed may be in the form of scroll or scan speed for theimages, for example images per second or minute, for example.

2. Streaming of Soundtrack to User Device from a Remote Server

General Overview

The invention primarily relates to the streamed delivery of asynchronised soundtrack to a user device from a remote server over anetwork. The soundtrack may be streamed in two main alternative forms,namely as a compiled soundtrack or an uncompiled soundtrack. In oneembodiment, a single user device is provided for both displaying thevisible electronic media content and receiving the streamed soundtrackfor audible playback. In another embodiment, a pair of separate userdevices is provided which are in data communication. One of the userdevices provides the visible electronic media content, while the otheruser device receives the streamed soundtrack for audible playback. Thedata communication between the devices enables the soundtrack playbackto be synchronised with the display of the electronic media content.

In the first form, when streaming as a compiled soundtrack, the serverstreams a compiled digital audio file of the soundtrack that is readyfor playback by any suitable generic or customised digital audio playersoftware, whether a stand-alone application or a plug-in or function ofanother application. A single digital audio file of the soundtrack maybe streamed or alternatively two or more audio files (each containingaudio regions of the soundtrack of a particular type, eg ‘music’ or‘effect’ or ‘ambience’, or some other breakdown) representing thesoundtrack may be streamed ready for playback concurrently together torecreate the entire soundtrack. Alternatively, the user may deactivateone or more of the audio files from playback to effectively mute one ormore types of audio regions from playback. The audio file or audio filesof the soundtrack delivered by the server are compiled at a configuredreading speed that may be selected by the user to suit their individualreading speed such that the playback timing of the audio regionscorresponds to their expected reading position within the electronicmedia content.

In the second form, when streaming as an uncompiled soundtrack, theserver streams the soundtrack data file and audio track files relatingto the audio regions of the soundtrack to the user device. The userdevice comprises a customised soundtrack playback module or functionthat is configured to process the soundtrack data file and co-ordinatesynchronised playback of the audio track files of the audio regions withthe user's estimated or expected reading position in the electronicmedia content. The soundtrack playback module or function may be astand-alone application or integrated with another application as aplug-in or the like. The soundtrack playback module is configured tomaintain a user reading position variable indicative of the user'sexpected reading position within the electronic media content andtrigger playback of the audio regions defined in the soundtrack datafile based on this user reading position variable, and may operate in asimilar manner to the soundtrack playback module described inWO2013/015694. By way of example, if the audio regions of the soundtrackare defined by start and stop positions relating to a word index orcount as described above, the user reading position variable may be acounter corresponding to the estimated word in the electronic text thatthe user is reading. The counter may then increment at a rate at leastpartly based on a configurable reading speed variable representing theuser's reading speed. The counter may also be arbitrarily orperiodically updated to the user's actual reading position by varioususer interaction or user input, including page turns, scrolling,arbitrary reading position update inputs such as the user clicking amouse cursor on the current word they are reading or tapping (in thecase of a touch screen display) on the current word they are reading, orother such means for updating the user reading position variable.

In both streaming cases above, for the compiled and uncompiledsoundtracks, the soundtrack may be streamed at a data transfer rate thatis equal to the data playback rate on the user device or more preferablyat a higher data transfer rate to account for data transfer delay orinterruptions caused by, for example, network latency, server or userdevice performance, or the like. When streaming at a higher datatransfer rate than the data playback rate, the user device is providedwith a buffer for temporarily storing the incoming data ready for futureplayback. The size of the buffer may be fixed or may dynamically changeduring streaming. The buffer size may be configurable and controlled byeither the user device or the server. If desired, the buffer size may beset to zero to effectively limit the data transfer rate to the dataplayback rate. Optionally, the system may be configured to vary the bitrate or quality of the soundtrack delivered to the user device based onthe bandwidth of the data channel between the server and user device.

The soundtrack may be streamed separately to or in combination with thecorresponding electronic media content, and the display and playback ofthe electronic media content and soundtrack may be controlled inseparate application programs or a combined application program inembodiments employing a single user device, or in separate applicationprograms running on separate devices in embodiment employing a pair ofuser devices in data communication. In some forms, the user may alreadyhave the electronic media content stored on a user device and onlyrequire the corresponding soundtrack from the server for playback onthat or a separate user device during reading of the electronic mediacontent. Referring to FIGS. 3A, 3B, and 4, some examples of thesedifferent system configurations will be described.

Example 1—Separate Applications on User Device for Text and Soundtrack

FIG. 3A shows a first example 30 of a system configuration in which theuser device 32 runs separate applications for controlling the display ofthe electronic media content and playback of the synchronised soundtrackreceived from the soundtrack server 34. In the system configuration 30,the user device 32 may comprise a first application 36 (an electronicmedia content or text application) which is responsible for theretrieval, processing and display of electronic media content 48 fromdata files stored locally or an accessible database or retrieved orstreamed from online or the internet or an electronic media contentserver (such as an electronic e-book server, store or library) on adisplay 38 of the user device. The text application 36 may be any formof electronic media content viewer or editor. In one example, the textapplication may be an e-reader software for displaying electronic mediacontent in the form of an e-book file. Alternatively, in another examplethe text application may be an Internet browser viewing electronic mediacontent in the form of a webpage.

The second application 40 (soundtrack application) is operable toreceive, process and playback an audio data stream or streamsrepresenting synchronised soundtrack for the electronic media contentreceived from a remote soundtrack server 34 over the network 42. Thesoundtrack application may be any software or application program thatis capable of interacting with the soundtrack server 34 via control data42 and which is capable of receiving one or more audio streams 44 fromthe soundtrack server 34, and which is further capable of processing theincoming audio data and playing the soundtrack over the audio outputsystem 46 of the user device 32. It will be appreciated that the audiooutput 46 may be any form of audio transducer, including integrated orexternal speakers, headphones, earphones or the like.

In a first form, text application 36 and soundtrack application 40 mayoperate independently without the exchange of data. In a second form,the applications 36, 40 may send or make data available to each othervia data communication channel or link 41, such an API or similar. Forexample, the text application 36 may send data to or make data availablefor retrieval by the soundtrack application 40, the data beingindicative of user reading information including, for example, user'scurrent reading position, reading speed, data indicative of page turns,scrolling information, arbitrary reading position updates, and/or dataindicative of the title or details identifying the item of electronicmedia content being read.

In one example, the soundtrack application 40 may be customisedsoundtrack player application which is configured for the sole purposeof accessing a synchronised soundtrack for an associated electronicmedia content from the remote soundtrack server 34. Alternatively, thesoundtrack application may be an add-on or plug-in to another main usersoftware application. In one such example, the soundtrack applicationmay be integrated with an Internet browser, and the soundtrackapplication may be a plug-in application to the Internet browser. Inanother example, the soundtrack application may be an Internet browserthat is able to access a soundtrack player webpage or soundtrack libraryhosted by the soundtrack server 34.

The typical operation of the first example system configuration 30 willnow be described. Firstly, the user loads electronic media content fordisplay via the text application 36. The user then launches theirsoundtrack application 40 and selects, or the application automaticallyselects based on the loaded electronic media content, the synchronisedsoundtrack corresponding to the electronic media content from thesoundtrack server 34. For example, the user may operate the soundtrackapplication 40 to access a soundtrack library 35 of the soundtrackserver which displays the available soundtracks for different electronicmedia content. The soundtrack library 35 represents all the soundtracksstored in a soundtrack database 37 that is integrated with or accessibleby the soundtrack server. For example if the user is displaying ane-book in the text application 36, then they may search the soundtrackserver via the soundtrack application 40 for the one or more associatedsoundtracks that have been customised or configured for that particulare-book. The user interacts with the soundtrack server via control data42 sent by their soundtrack application 40 in response to userinput/interaction. Once the soundtrack has been selected, the server 34,via audio streaming module 39, may stream the soundtrack via one or moreaudio data streams 44 to the user device 32 for playback by thesoundtrack application 40.

Compiled Soundtrack Streaming

If the server is configured to stream a compiled soundtrack, a userreading speed setting is configured for the user and this may be storedin a user reading information module 31. For example, the user readingspeed setting may be configured in various ways via the soundtrackapplication 40 and sent to the server via the control data 42, includingthe user inputting a numerical reading speed, selecting a reading speedfrom a list, or a scale, conducting a reading speed test provided by thesoundtrack server via the soundtrack application 40, or the user'sreading speed may be extracted from historical data stored in thesoundtrack application 40 or in the soundtrack server 34, for example.Once the user reading speed setting is configured, the server isconfigured to stream a compiled soundtrack that corresponds to the userreading speed setting. As will be explained in further detail later, thecompiled soundtrack may be compiled or progressively compiled inreal-time to a specific reading speed prior to streaming oralternatively a library of pre-compiled soundtracks at a range ofdifferent reading speeds may be available in the server and the one mostclosely matching the user's reading speed setting may be selected(automatically by the server or manually by the user) to be streamed tothe user device. The soundtrack application 40 stores or buffers theincoming audio data stream into memory, and the user may initiateplayback of the ready-to-play audio file of the soundtrack via thesoundtrack application as they commence reading, or playback can beautomatically initiated by the soundtrack application 40 based on datafrom the text application 40 indicative of the user's interaction withthe electronic media content.

Uncompiled Soundtrack

If the server is configured to stream an uncompiled soundtrack, thesoundtrack data file and audio track files defining the audio regionsare streamed to the user device 32 over one or more audio data streams44 by the server 34 in response to control data 42 from the soundtrackapplication 40 on the user device. The soundtrack application 40 storesor buffers the incoming audio data stream into memory. As will beexplained in further detail later, the soundtrack data file may be sententirely or in predefined chunks or sections. Likewise, the audio trackfiles may be sent one by one in order of playback or in batchescorresponding to predefined chunks or sections of the soundtrack. Theuser configures their reading speed setting (either manually or it isautomatically detected) in the soundtrack application 40 on the userdevice using any of the previously discussed techniques. The user mayinitiate playback of the soundtrack as they commence reading or this canoccur automatically as above, which causes the soundtrack application toprocess the soundtrack data file and trigger the co-ordinated playbackof the audio track files corresponding to the audio regions based on auser reading position variable indicative of the user's estimated orexpected reading position in the electronic media content as previouslyexplained.

Example 2—Single Application on User Device for Text and Soundtrack

FIG. 3B shows a second example of another system configuration 60 inwhich the user device 32 runs a single application (multimediaapplication) to control the display of the electronic media content andplayback of the synchronised soundtrack, but is otherwise similar to thefirst system configuration 30. Like reference numerals represent likecomponents and functionality in the drawings. In this example, a singlemultimedia application 62 is configured to run on the user device 32 andcontrols the display of the electronic media content and the playback ofthe soundtrack, functionality that was carried out by the text andsoundtrack applications 36, 40 in the first system configuration 30.

In this system configuration 60 the multimedia application 62 may beconfigured to interact with a multimedia content sever 64 to withcontrol data 42 to receive the audio stream 44 representing thesoundtrack synchronised to the electronic media content viewed on theuser device in a process similar to that described with respect to thefirst system configuration 30.

The multimedia application 60 may source the electronic media contentfrom locally stored data files or data files stored in an accessibledatabase or retrieved or streamed from online or the internet or anelectronic media content server (such as an electronic e-book server,store or library). In one example, the multimedia data application mayfully download, progressively download, or stream the electronic mediacontent from the multimedia content server 64 over a electronic mediacontent data channel 66. In such examples, the multimedia content server64 may comprise an electronic media content module 68 that is configuredto control the delivery of the electronic media content to the userdevice over the network 43.

In one example, the multimedia application 62 may be a custom softwareapplication program that is configured for interaction with themultimedia content server and which enables the user to selectmultimedia content comprising electronic media content and an associatedsynchronised soundtrack for display and playback respectively from themultimedia content server. By way of example, the multimedia contentserver may provide a library 70 of e-books and correspondingsynchronised soundtracks customised for the e-books. The e-book data maybe sent in the electronic media content data stream 66 and thesoundtrack may be streamed in the audio data stream 44 to the userdevice. Alternatively, the electronic media content and audio datastreams need not be separate and may be combined into a singlemultimedia data stream.

In another example, the multimedia application 62 may be in the form ofan Internet browser. In such examples, the multimedia content server maybe a website server that in the electronic media content may be awebpage downloaded to the Internet browser over electronic media contentdata stream 66. In parallel, the Internet browser may have a soundtrackplayer functionality as an plug-in application or alternatively asoundtrack player functionality may be an embedded application withinwebpage content which is configured to control the audio streamstreaming 44 for the and soundtrack playback.

The soundtrack application functionality of the multimedia application62 operates similarly to the first system configuration 30 as above, inthe context of streaming either compiled or uncompiled soundtracks. Asthe multimedia application 60 has inherent knowledge about the user'sinteraction with the electronic media content, it can determine updateduser reading positions, periodically or arbitrarily, based on userinteraction/input (e.g. page turns, scrolling, arbitrary readingposition update inputs etc) to assist in co-ordinating synchronisedplayback of the compiled or uncompiled soundtrack as will be explainedin further detail later.

Example 3—Pair of User Devices in Data Communication

FIG. 4 shows a third example of another system configuration 80 in whicha pair of user devices 82, 84 is provided. The first user device 84(audio playback device) has an audio output 85 and is configured toprovide the audible soundtrack to the user, and the second user device82 (text display device) has a display 83 and is configured to providethe user with the visible electronic media content. This third systemconfiguration 80 is similar to the first system configuration 30 of FIG.3A, except the text application 36 and soundtrack application 40 havebeen separated to operate on separate user devices. Like referencenumerals represent like components and functionality in the drawings,and won't be repeated.

By way of example, the data communication link 86 between the pair ofuser devices 82, 84 may be, direct or indirect, including via RadioFrequency (RF), Near-field communication (NFC), Bluetooth, infrared,WiFi or other such wireless communication mediums or protocols, oralternatively a wired communication medium may be provided if the pairof user devices are connected via a cable or cables. The data connection86 between the user devices 82, 84 serves a similar purpose to the datacommunication channel 41 described in the system configuration 30 ofFIG. 3A. In particular, the text display device 82 may be configured tosend data to or make data available for retrieval by the audio playbackdevice 84 over the data link 86, the data being indicative of userreading information including, for example, user's current readingposition, reading speed, data indicative of page turns, scrollinginformation, arbitrary reading position updates, and/or data indicativeof the title or details identifying the item of electronic media contentbeing read.

The electronic media content data for display on the text display device82 may be stored locally on the device or an accessible database or, asshown, the data 87 may be downloaded, streamed or otherwise retrievedfrom an electronic media content server 88 over a network 89. Theelectronic media content server 88 may be in the form of an e-bookserver, store or library, a website server, or any other form ofelectronic media content server such as, for example, one having anelectronic media content module 68 and electronic media content library70 as described with reference to the system configuration 60 in FIG.3B, where like numerals represent like components.

As an example of the system configuration 80, the text display device 82might be in the form of an e-Reader device, tablet or smartphone runningan e-Reader application or other text application, desktop computer orany other electronic device capable of running a text application 36 anddisplaying the electronic media content on the display 83. Likewise, theaudio playback device 84 may be in the form of a tablet, smartphone,computer, dedicated audio playback device, or any other electronicdevice capable of running the soundtrack playback application 40 togenerate the audible playback of the audio output 85. The user mayoperate any pair of such devices which are in data communication, toview the electronic media content on one device 82, while hearing thesynchronised soundtrack via the other device 84.

Server Configurations for Electronic Media Content

As described above in the various system configurations 30, 60, 80above, the electronic media content data, in some embodiments, may bestreamed, downloaded or otherwise retrieved from the same server as thesoundtrack, or alternatively a separate server, e.g. a 3^(rd) partyserver. In particular, in some forms a multimedia server is providedwhich provides both the soundtrack and electronic media content, and inother forms a soundtrack server is provided and an electronic mediacontent server is provided. In the case of separate servers, each servermay be owned and operated by the same entity or separate entities.

Various embodiments or implementations of the soundtrack delivery systemwill now be described with reference to FIGS. 5A-7. In particular,example embodiments for streaming compiled soundtracks will be describedwith reference to FIGS. 5A-6 and example embodiments for streaminguncompiled soundtracks will be described with reference to FIG. 7. Theseimplementations can apply to any one of the system configurations ofFIGS. 3A-4. Where a single user device system configuration like FIGS.3A and/or 3B is described, it will be appreciated that this may beadapted or modified to operate on a pair of user devices as describedfor the system configuration of FIG. 4.

3. Streaming of Pre-Compiled Soundtracks Overview

Referring to FIG. 5A, a first form of the soundtrack delivery system 100is configured to stream a pre-compiled soundtrack to the user device 102from the soundtrack server 104. The soundtrack server has an integratedor an accessible soundtrack database 106 comprising a range ofpre-compiled soundtrack audio files that are ready-to-play, eachcompiled for a different user reading speed. The soundtrack server isconfigured to automatically select or the user may manually select apre-compiled soundtrack audio file that most closely matches the user'sreading speed and deliver that to the user device 102 over the network108.

Hardware

The user device 102 may be any programmable electronic device.Typically, the user device has a processor 110, memory 111, a userinterface 112 such as keyboard, mouse, touch display screen, controlbuttons, voice activation, or similar, a display or display screen 113,such as an LED display or similar, a communication module 114 forconnecting to the soundtrack server 104 over a network 108 and an audiooutput 115 which can deliver sound to the user via integrated orexternal speakers, headphones, earphones or similar. The communicationmodule may communicate with the soundtrack server over a network via anywired or wireless communication mediums, or a combination of these. Forexample, for smartphones the communication module may be the 3G or 4Gcellular data connection, or alternatively a Wi-Fi connection to abroadband router. In the context of a personal computer, thecommunication medium may be hardwired or wireless connection to a localarea network (LAN).

System Operation

In operation, the user device may be operable to access the soundtrackfrom the soundtrack server 104 via a soundtrack application or amultimedia application running on the user device as described withreference to FIGS. 3A-4. As previously described, the electronic mediacontent may be either streamed from the soundtrack server, a separate3^(rd) party server, or alternatively loaded locally by the user devicein alternative configurations. For clarity, the description will focuson the soundtrack streaming aspect.

Soundtrack Selection

The user may select a soundtrack manually for the electronic mediacontent from the soundtrack library on the soundtrack server whichdisplays the soundtrack data files 116 available on the server.Alternatively, the soundtrack data file for the electronic media contentmay be automatically selected by the soundtrack server if it is alsoproviding electronic media content or if the downloaded or stored mediacontent from a third party server is known. The user device interactswith the soundtrack server via the control data stream 118.

A user reading speed setting may be then configured for the user. Theuser reading speed setting may be sent from the user device to thesoundtrack server over the control data stream 118. Alternatively, thesoundtrack server may store a user reading speed for the user based onhistorical user reading speed information from previous interactions.

In one form, the soundtrack server may comprise a soundtrack selectormodule 120 that is configured to maintain a user reading speed variablethat represents the user's reading speed. The soundtrack selector moduleis configured to search the soundtrack database for a pre-compiledsoundtrack audio file that most closely matches the user reading speedvariable. In particular, the soundtrack database 106 comprises a libraryof soundtracks pre-compiled at different user reading speeds for eachsoundtrack data file 116 in the soundtrack library. By way of exampleonly, the soundtrack database may comprise the soundtrack audio filesthat have been pre-compiled at reading speeds of say 200 words perminute (wpm), 250 wpm, 300 wpm, 350 wpm, 400 wpm, etc. Each of thesepre-compiled soundtrack audio files will have a different duration thatcorresponds to the user reading speed for which it was compiled. Thefaster reading speeds having a shorter duration audio file than theslower reading speeds. Each of the pre-compiled soundtrack audio files122 has a reading speed identifier that represents the reading speed forwhich it was compiled or produced. This reading speed identifier may bein the soundtrack file name, embedded or encoded in an associated headerfile or information of the audio file or may otherwise be stored in adata or folder structure that identifies the pre-compiled reading speed.The soundtrack selector 120 is configured to determine whichpre-compiled soundtrack audio file 122 has a reading speed identifier orwas pre-compiled at a reading speed that most closely matches the userreading speed variable. That selected soundtrack is then queued forstreaming to the user device via the audio streaming module 124 of thesoundtrack server.

In an alternative form, the user may simply review the pre-compiledsoundtrack audio files in the database and select the soundtrack with areading speed identifier to which they would like to receive withoutconfiguring a user reading speed variable in the soundtrack server.

Soundtrack Streaming

Once the pre-compiled soundtrack audio file 122 is selected, the audiostreaming module 124 commences streaming of the pre-compiled soundtrackaudio file on the audio data stream 126 over the network 108 to the userdevice 102. The audio stream 126 may commence at the start of thepre-compiled audio track or at an offset position within thepre-compiled soundtrack audio file corresponding to either the user'sdesired commencement reading position within the electronic mediacontent. In one form, the user device 102 is configured to send controldata 118 representing the user's desired commencement reading positionin the electronic media content to the server 104 such that the incomingaudio stream commences at that position within pre-compiled soundtrackaudio file.

At the user device, the incoming audio stream 126 is received and storedtemporarily in a buffer in memory ready for playback. The buffer sizemay be configured by the user device or alternatively by the soundtrackserver, for example via header information provided in the audio stream.The buffer size may be configured or varied to suit the processing anddata connection speed of the user device for example. If the bufferfills up or reaches full capacity, control data 118 is sent from theuser device to the soundtrack server to halt the data transfer on theaudio stream, until reinitiated by the user device when capacity becomesavailable in the buffer.

When the user begins reading the electronic media content, they mayinitiate playback over the audio output via direct control of thesoundtrack player of the soundtrack application or multimediaapplication. By way of example, the soundtrack playback may be triggeredvia user interaction with soundtrack or multimedia application, such asgenerating a reading commencement signal by operation of a play buttonor similar in a graphical user interface (GUI) of the soundtrack ormultimedia application. Likewise, the user may pause soundtrack playbackvia a pause button or similar in the GUI. Alternatively, the soundtrackplayback initiation and pausing may be automatically controlledindirectly via the user's interaction with the electronic media contentwhich may signal user reading commencement or pausing. In suchembodiments, the soundtrack player may be a hidden background processnot directly controllable by the user.

In one form, the buffer size or range is configurable relative to thecurrent playback position within the audio stream. It may be as small oras large as the user desires within the capabilities of their userdevice, each having trade-offs. A small or no buffer may result in lessdata usage (in the context of mobile or broadband data allocationsubscriptions) but higher likelihood of an interrupted or discontinuoussoundtrack playback due to network delays and/or latency and no orminimal ability to re-synchronise to a new reading position quickly. Alarger buffer provides a higher likelihood of a continuous soundtrackplayback and more ability to re-synchronise to a new reading positionquickly, but can result in higher data usage. The buffer extends forwardof the current playback position and may optionally also extend behindthe current playback position such that a played portion of the audiostream is retained in memory for a period of time. The buffereffectively is a sliding buffer about a reference point determined bythe current playback position of the soundtrack within the audio stream.

Re-Synchronising Soundtrack to a New Reading Position

Referring to FIGS. 5A and 5B, the user may operate the user device togenerate data indicative of a new user reading position within theelectronic media content that the user is reading, or this may beautomatically detected by the user device based on user interaction. Byway of example, the user may indicate a new reading position based onuser interaction/input such as page turns, scrolling, arbitrary readingposition update inputs, etc. For example the user may jump or skip to anew page or portion of the electronic media content and begin readingfrom a new reading position.

Upon detecting a new reading position 130, the soundtrack application ormultimedia application running on the user device is configured todetermine whether the new user reading position is within the range ofthe buffered portion of the streamed soundtrack in memory 131.

If the new reading position is within the buffer range 132, then apre-configured or configurable playback position transition is initiated133. Various transition options are available. In one example, an abrupttransition 134 may be implemented such that the soundtrack playbackabruptly skips or jumps to the new playback position and continuesplayback from that new position. In another example, a halt transition135 may be implemented whereby the current playback is halted, and thesoundtrack is cued for playback at the new position but awaits a userinitiation (e.g. activation of a play button or similar) beforere-commencing playback at the new position. In another example, a fadetransition 136 may be implemented in which the current soundtrackplayback position may progressively fade out with the new soundtrackplayback position progressively fading in, like a cross-fade oralternatively the new soundtrack could be faded in only once the currentsoundtrack has been completely faded out.

If the new reading position is outside the buffered audio stream 137,such as likely when the user skips many pages or chapters of an e-bookfor example, the soundtrack application or multimedia applicationrunning on the user device may be configured to send control data 118 tothe soundtrack server indicative of the new user reading position 138and this may be optionally stored in a reading position module 109 inthe server. In response, the audio streaming module resets the audiostream by halting the current data transfer 139 and re-commencing datatransfer of the pre-compiled soundtrack at the new playback positionthat corresponds to the new current reading position. Upon receiving thenew audio stream, the user device is configured to implement a streamtransition protocol 140. Various transition options are available. Inone example, an abrupt transition 141 may be implemented in which thesoundtrack playback immediately jumps to the new audio stream uponarrival at the user device. In another example, a halt transition 142may be performed in which the playback of the current stream is halted,the buffer cleared, and the new audio stream is buffered into the userdevice awaiting playback initiation by the user. In another example, afade transition 143 may be implemented in which the current audio streamis progressively faded out while the new soundtrack is progressivelyfaded in, like a cross-fade or the new audio stream may be faded in onlyonce the current audio stream has been completely faded out or the twostreams are matched by beat/tempo/content type so that the cross fadefrom old to new stream is seamless.

Changing Reading Speeds—Transition Protocols

The user's reading speed may increase or decrease during reading of theelectronic media content. The soundtrack application or multimediaapplication on the user device may receive manual updates of changes tothe user's reading speed, via the user selecting a new reading speedfrom a range or entering a new numerical reading speed or other userinput indicative of a change in reading speed. Alternatively, themultimedia application which controls the display of the electronicmedia content and the soundtrack playback may automatically detectsignificant changes in the user reading speed.

Referring to FIGS. 5A and 5C, if a change in reading speed is detected150, control data 118 indicative of the new reading speed is sent to thesoundtrack server from the user device. The soundtrack server isconfigured to determine whether the currently selected pre-compiledsoundtrack audio file should transition to another pre-compiledsoundtrack audio file that more closely matches the new user readingspeed. If a new pre-compiled soundtrack audio file is selected 151 forstreaming, then a transition protocol 152 is executed by the soundtrackserver to transition playback to the new pre-compiled soundtrack audiofile. Various examples of possible transition protocols will bedescribed in further detail for switching the synchronised soundtrackbetween a current and a new user reading speed. It will be appreciatedthat other transition protocols may also be implemented.

Abrupt Transition

In one example, the transition protocol may be an abrupt transition aswill be described further with reference to the steps 152A. In theabrupt transition protocol, the user's current reading position in theelectronic media content is determined at step 154. This may bedetermined via control data 118 sent from the user device to thesoundtrack server for example. The soundtrack server then commences anew audio stream 156 corresponding to the new pre-compiled soundtrackaudio file selected for the updated reading speed. The playback positionof the new audio stream within the new pre-compiled soundtrack audiofile corresponds to the user's current reading position as determined atstep 154. The new audio stream commenced at step 156 is in parallel tothe current audio stream for the old reading speed. At the user device,upon receiving the new incoming audio stream, the user device clears thebuffer and abruptly switches to the new audio stream and commencesplayback at step 158. Finally, the old audio stream is halted at 160 bythe server.

Fade Transition

In another example, the soundtrack server may implement a fadetransition protocol as will be described with reference to steps 152B.In the fade transition protocol, steps 154, 156 and 160 are the same ascarried out for the abrupt transition 152A. The primary difference isstep 162 where the user device implements a fade switch to the newincoming audio stream. For example, upon receiving the new incomingaudio stream, the user device is configured to progressively fade outthe old audio stream, while concurrently fading in the new audio stream.Alternatively the old audio stream may be completely faded out beforethe new audio stream commences fade in.

Timed Transition Protocol

In another example, the soundtrack server may implement a timedtransition protocol as will be described with reference to reference tosteps 152C. In this timed transition protocol, the soundtrack serverconducts an analysis of the remaining unstreamed (i.e. the data relatingto the portion of the audio file that has not yet been transmitted fromthe server to the user device) portion of the pre-compiled soundtrackaudio file in the current stream for an appropriate transition position.In one example, the soundtrack server may perform signal processinganalysis of the unstreamed portion of the audio file to determine a lullor gap or period of silence. Alternatively, the server may analyse thecorresponding soundtrack data file which contains the timing data forthe audio regions in the pre-compiled audio file, and may analyse orcalculate a next appropriate transition position based on presettransition parameters or criteria. By way of example, the transitioncriteria may dictate that the only appropriate transition positions arethose in the soundtrack where there are no audio regions correspondingto effects and/or ambience, or alternatively no audio regions playing atall or through matching beat, tempo, or content type of audio tracks.Various transition criteria could be configured depending on the natureof the soundtrack including no transition or a default transition if nocriteria can be easily or quickly met within a specified timeframe. Theabove transition analysis is performed at step 164. Once the nexttransition position is determined, the soundtrack server is configuredto switch in the new pre-compiled audio file into the audio stream datatransfer in place of the current audio file when the audio data relatingto the transition position is reached 166. At the user device, aseamless transition from the old audio file at the old reading speed tothe new audio file at the updated reading speed is experienced usingaudio techniques such as fade in, fade out, volume and equalizermatching and other audio processing techniques, unbeknown to the user.

4. Streaming of Real-Time Compiled Soundtracks Overview

Referring to FIG. 6, a second form of the soundtrack delivery system 200is configured to stream a real-time compiled soundtrack to the userdevice 102 from the soundtrack server 204. In this system configuration200, the soundtrack server comprises a library of soundtrack data files116 and an integrated or accessible soundtrack database 206 comprisingthe audio track files associated with the audio regions defined for allthe soundtrack data files. In this system configuration 200, thesoundtrack server 204 is provided with a real-time soundtrack compiler220 that is configured to process the soundtrack data file for aselected soundtrack and to progressively compile or generate a compiledsoundtrack audio file for streaming to the user in real-time or ondemand. The soundtrack is compiled according to a configurable userreading speed setting corresponding to the user's reading speed.

As will be explained, the operation of the second form soundtrackdelivery system 200 is substantially similar to that of the first formsoundtrack delivery system 100, and like reference numerals representlike components and functionality in the drawings. The primarydifference with the second form system 200 is that the compiledsoundtrack is compiled or generated in real-time for a configured userreading speed from the selected soundtrack data file and audio trackfiles associated with the audio regions defined in the soundtrack datafile, as opposed to the soundtrack audio file being entirelypre-compiled as in the first form system 100. Otherwise, the systemoperation relating to the soundtrack selection, audio streaming,re-synchronisation to a new reading position, and transition protocolsfor changing reading speeds is substantially similar, as will be furtherexplained.

Real-Time Compiling and Soundtrack Streaming

Once a soundtrack is selected, the user device sends control data 218 tothe soundtrack server 204 indicative of the user's reading speed and adesired commencement reading position within the electronic mediacontent. Alternatively, the server 204 may retrieve the user's readingspeed setting from historical stored data. The real-time soundtrackcompiler 220 then commences compiling or production of an audio filerepresenting the soundtrack commencing at the desired reading positionand with audio region playback timing that corresponds to the user'sconfigured reading speed setting. The soundtrack server is configured toprogressively begin streaming the partially built or produced audio fileof the soundtrack as it is being compiled via the audio streaming module224 to the user device. In particular, the compiled portion of thesoundtrack may be streamed simultaneously while the soundtrack compileris compiling the remaining portion of the audio file in real-time to thecustomized user reading speed setting.

In one embodiment, the soundtrack compiler may produce a single audiofile which is sent on a single audio stream 226 to the user device 102.In an alternative embodiment, the soundtrack compiler may produce two ormore separate audio files 228 representing the soundtrack. For example,the soundtrack compiler may progressively produce a music audio filecontaining all the music audio regions, a sound effects audio filecontaining all the sound effects audio regions, and an ambience audiofile containing all the ambience audio regions. All these audio fileswill have the same duration and may all be played back concurrently atthe user device to recreate the soundtrack or a selection of the audiofiles may be played back concurrently to create a modified soundtrack inwhich only audio regions of a certain type are heard, with the othersmuted.

The extent to which the soundtrack compiler 320 compiles or produce thesoundtrack in advance beyond that transferred to the user device may beconfigured or varied based on a variable compilation setting. Forexample, for faster readers the soundtrack compiler may compile more ofthe audio file in advance for streaming compared to slower readers.Alternatively, or additionally, control data 218 from the user devicemay control the amount of soundtrack being compiled in advance of thecurrent portion transferred to the user device. For example, controldata 218 from the user device indicating that the user's buffer is full,may cause the soundtrack compiler to pause compiling any furtherremaining portion of the audio file or files until receiving furthercontrol data indicating that the buffer has capacity.

Soundtrack Playback and Transitioning

Upon receiving the incoming audio stream or streams of the real-timecompiled audio file or files, the user device buffers the audio datainto memory and may commence and pause playback as previously describedwith reference to the first form system 100.

In regard to transitioning to a new reading position, the same sequenceof operations as described with reference to FIG. 5B substantially applyin regard to the first form system 100, except with a few adjustments.In particular, if the new reading position is outside the buffer range,at step 139 the server may either reset the audio stream immediately tothe new playback position if it is within the compiled but yetunstreamed portion of the audio file, or alternatively if the newplayback position is beyond what has been compiled then the soundtrackcompiler is configured to re-commence compiling of the soundtrack at thenew position and then reset the audio stream to being transferring thenew compiled portion of the audio file or files of the soundtrack. Astream transition protocol 140 may then be implemented as described inwith reference to FIG. 5B.

In regard to changing reading speeds, the same sequence of operations asdescribed with reference to FIG. 5C substantially apply in regard to thefirst form system 100, except with a few adjustments. When a new readingspeed is detected 150, rather than the soundtrack server selecting a newpre-compiled soundtrack for streaming 151 as in the first form system100, the soundtrack server in the second form 200 is configured toinitiate the soundtrack compiler 220 to begin compiling a new audio fileor files of the soundtrack is configured at the new reading speed. Atransition protocol 152 for this new audio file or files of thesoundtrack at the new reading speed may then be implemented as describedwith reference to FIG. 5C.

5. Streaming of Uncompiled Soundtracks Overview

Referring to FIG. 7, a third form of the soundtrack delivery system 300is configured to stream an uncompiled soundtrack to the user device 102from the soundtrack server 304. In this system configuration 300, thesoundtrack server 304 comprises a library of soundtrack data files 316and an integrated or accessible soundtrack database 306 comprising theaudio track files associated with the audio regions defined for all thesoundtrack data files. In operation, the soundtrack server is configuredto stream, via the audio streaming module 324, the soundtrack data fileand the audio track files to the user device 102 (like referencenumerals representing like components and functionality previouslydescribed) over one or more audio data streams. The soundtrackapplication or multimedia application on the user device is providedwith a soundtrack player 330 that is configured to process thesoundtrack data file and control timed playback of the audio track filescorresponding to the defined audio regions based on a user readingposition variable indicative of the user's current reading position inthe electronic media content.

Soundtrack Streaming

The user may select the soundtrack for streaming or this may beautomatically selected by the soundtrack server if it is aware of theelectronic media content being viewed by the user device, for example inconfigurations where the soundtrack server is in the form of aelectronic media server providing the electronic media content incombination with the synchronized soundtrack. This interaction with thesoundtrack server may occur via control data 318.

Once the soundtrack for streaming is selected, the entire soundtrackdata file is firstly sent to the user device on the audio data stream326. The soundtrack data file 332 is then processed by the soundtrackplayer. In particular, the soundtrack player comprises an audio trackfile requester module 331 that controls the progressive download of theaudio track files associated with the soundtrack from the soundtrackserver. The audio track file requester module 331 sends control data 318to the audio track file dispatcher module 320 of the server requestingthe required audio track files. In response to these requests, thedispatcher module 320 retrieves the requested audio track files from thesoundtrack database and sends them to the user device on the audio datastream. The soundtrack player stores the requested audio track files inmemory, ready for playback.

The soundtrack player comprises a reading position module 333 thatmaintains a user reading position variable (such as a counter aspreviously described) that is indicative of the user's expected orestimated reading position in the electronic media content. This userreading position variable increments or changes at a rate based on aconfigured user reading speed setting, and may be updated periodicallyor arbitrarily based on user interaction/input (e.g. page turns,scrolling, arbitrary reading position update inputs etc). The readingposition module is also configured to receive control input indicativeof the user commencing reading or pausing reading, and controls thereading position variable accordingly (eg starts or pauses the counter).

In a first form, the requester module 331 controls the progressiveretrieval of the audio track files based on the user reading positionvariable and the downloaded soundtrack data file. In one example, therequester module processes the soundtrack data file to determine eachnext audio region due for playback based on the user reading positionvariable and requests the dispatcher module 320 in the server to sendthe audio track file for that next audio region on the audio data streamto the user device in advance ready for playback. This may be done oneby one for each audio region or alternatively in batches or chunks ofaudio track files may be requested in advance for the next batch orchunk audio regions due for playback based on the user reading positionvariable. In another example, the requester module may process thesoundtrack data file and maintain a sliding download window relative tothe user reading position variable. In particular, the sliding downloadwindow may be a configurable size that extends from the user readingposition variable. The requester module 331 may be configured to ensureit has requested all audio track files corresponding to any audioregions beginning within the sliding download window. The larger thewindow, the more audio track files are downloaded in advance of theirplayback and this reduces the likelihood of soundtrack playbackinterruption due to network delays.

In a second form, the requester module 331 may be configured to downloadall the audio track files associated with the soundtrack from theserver, one by one in the order of the playback of their associatedaudio regions.

In a third form, the soundtrack may be divided into multiplepre-determined sections or portions, and this information may beembedded in the soundtrack data file. For example, in the context of asoundtrack for an e-book, the soundtrack may be divided into sectionscorresponding to each chapter of the e-book. In this form, the requestermodule 331 is configured to download the batch of audio files for eachsection entirely in advance of playback of that section. The requestermodule 331 may optionally be configured to also download the batch ofaudio files for one or more of the subsequent sections in advance duringconcurrent playback of the first downloaded section of audio files.

Soundtrack Playback

The soundtrack player 330 comprises a playback module 334 that isconfigured to process the soundtrack data file and trigger or cueplayback of the audio track files associated with the defined audioregions of the soundtrack based on the user reading position variablemaintained by the reading position module 333. In particular, theplayback module commences playback of the audio track file associatedwith an audio region when the reading position variable matches thestart position defined for that audio region, and stops playback of theaudio track file when the reading position variable matches the stopposition defined for that audio region, in a manner previouslyexplained. It will be appreciated that two or more audio track files maybe played concurrently when there are two or more overlapping audioregions and that each audio region can have other associated propertiesfor playback that may or may not be dependent on reading speed such asvolume and panning envelope control and other audio effects andproperties, as described in WO2013/015694.

Purging and Retention Policy

In this embodiment, the requester module 331 is configured to deleteaudio track files stored in memory either after they are played or aftera configurable predetermined time period, in case the user skips back tore-play a portion of the soundtrack. Additionally, the requester module331 is configured to process the soundtrack data file to determine anyaudio regions that define the same audio track files. Such repeatedaudio track files are flagged for retaining in the memory of the userdevice until the last audio region has played.

Sourcing Audio Track Files from Different Sources

In the above examples, the requester module 331 is configured to requestall the audio track files from the soundtrack server 304. In alternativeembodiments, the soundtrack data file from the soundtrack server 304 maybe configured to define one or more different sources or servers for atleast some or all of the audio track files. In such embodiments therequester module 331 may download, stream or source the audio trackfiles from the soundtrack server in combination with any other one ormore servers or file sources or file services, whether remote from theuser device or a music library or database stored locally on the userdevice, or alternatively may source all the audio track files from oneor more servers or file sources or file services that are distinct andseparate to the soundtrack server 304 providing the soundtrack datafile. In one example, the user device, in accordance with the soundtrackdata file, may obtain the bulk of the audio track files for audioregions relating to effects and ambience from the soundtrack server, andthe audio track files for the music audio regions from a different musicserver or their own music library or database stored on the user device.In another example, the user device, in accordance with the soundtrackdata file, may obtain all of the audio track files of the soundtrackfrom one or more sources separate to the soundtrack server 304, such asa file server, library or database accessible by or stored locally onthe user device. The alternative sources to the soundtrack server 304may be operated by the user or a different entity to the operator of thesoundtrack server 304.

6. General

Embodiments of the invention may be implemented by hardware, software,firmware, middleware, microcode, or any combination thereof. Whenimplemented in software, firmware, middleware or microcode, the programcode or code segments to perform the necessary tasks may be stored in amachine-readable medium such as a storage medium or other storage(s). Aprocessor may perform the necessary tasks. A code segment may representa procedure, a function, a subprogram, a program, a routine, asubroutine, a module, a software package, a class, or any combination ofinstructions, data structures, or program statements. A code segment maybe coupled to another code segment or a hardware circuit by passingand/or receiving information, data, arguments, parameters, or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

In the foregoing, a storage medium may represent one or more devices forstoring data, including read-only memory (ROM), random access memory(RAM), magnetic disk storage mediums, optical storage mediums, flashmemory devices and/or other machine readable mediums for storinginformation. The terms “machine readable medium” and “computer readablemedium” include, but are not limited to portable or fixed storagedevices, optical storage devices, and/or various other mediums capableof storing, containing or carrying instruction(s) and/or data.

The various illustrative logical blocks, modules, circuits, elements,and/or components described in connection with the examples disclosedherein may be implemented or performed with a general purpose processor,a digital signal processor (DSP), an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA) or otherprogrammable logic component, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general purpose processor maybe a microprocessor, but in the alternative, the processor may be anyconventional processor, controller, microcontroller, circuit, and/orstate machine. A processor may also be implemented as a combination ofcomputing components, e.g., a combination of a DSP and a microprocessor,a number of microprocessors, one or more microprocessors in conjunctionwith a DSP core, or any other such configuration.

The methods or algorithms described in connection with the examplesdisclosed herein may be embodied directly in hardware, in a softwaremodule executable by a processor, or in a combination of both, in theform of processing unit, programming instructions, or other directions,and may be contained in a single device or distributed across multipledevices. A software module may reside in RAM memory, flash memory, ROMmemory, EPROM memory, EEPROM memory, registers, hard disk, a removabledisk, a CD-ROM, or any other form of storage medium known in the art. Astorage medium may be coupled to the processor such that the processorcan read information from, and write information to, the storage medium.In the alternative, the storage medium may be integral to the processor.

One or more of the components and functions illustrated the figures maybe rearranged and/or combined into a single component or embodied inseveral components without departing from the invention. Additionalelements or components may also be added without departing from theinvention. Additionally, the features described herein may beimplemented in software, hardware, or combination thereof.

In its various aspects, the invention can be embodied in acomputer-implemented process, a machine (such as an electronic device,or a general purpose computer or other device that provides a platformon which computer programs can be executed), processes performed bythese machines, or an article of manufacture. Such articles can includea computer program product or digital information product in which acomputer readable storage medium containing computer programinstructions or computer readable data stored thereon, and processes andmachines that create and use these articles of manufacture.

The foregoing description of the invention includes preferred formsthereof. Modifications may be made thereto without departing from thescope of the invention as defined by the accompanying claims.

What is claimed is:
 1. A method of streaming a compiled soundtrack to aremote user device for a reader of electronic media content, thesoundtrack defined by multiple audio regions, each audio region definedby: an audio track for playback in the audio region, a start position inthe electronic media content corresponding to where the playback of theaudio region is to begin, and a stop position in the electronic mediacontent corresponding to where the playback of the audio region is tocease, the method comprising: receiving from a remote user device over anetwork control data indicative of the user's reading speed; andstreaming a compiled soundtrack to the user device over the networkbased on the received control data.
 2. The method according to claim 1wherein the compiled soundtrack is a pre-compiled soundtrack and isselected from a range of pre-compiled soundtracks that have beenpre-compiled at different reading speeds.
 3. The method according toclaim 2 wherein the method further comprises selecting the pre-compiledsoundtrack for streaming having an associated reading speed that mostclosely matches the control data indicative of the user's reading speed.4. The method according to claim 1 wherein the compiled soundtrack is areal-time compiled soundtrack and the method further comprises compilingthe soundtrack in real-time based on the control data indicative of theuser's reading speed.
 5. The method according to claim 4 wherein thesoundtrack is compiled progressively and the method comprises streamingthe compiled portion of the soundtrack while the remaining portion isbeing compiled.
 6. The method according to claim 1 wherein the methodfurther comprises receiving control data from the remote user deviceindicative of the user's desired commencement reading position andcommencing streaming of the compiled soundtrack to the user device at adata position within the compiled soundtrack that corresponds to theuser's desired commencement reading position.
 7. The method according toclaim 6 wherein the data position is at the start of the compiledsoundtrack or at an offset position within the compiled soundtrack. 8.The method according to claim 1 wherein the method further comprisesreceiving control data from the remote user device indicative of achange in reading position of the user to a new reading position in theelectronic media content, and modifying the streaming of the compiledsoundtrack based on the received control data.
 9. The method accordingto claim 8 wherein modifying the streaming of the compiled soundtrackcomprises adjusting the streaming of the compiled soundtrack to a newdata position within the compiled soundtrack that corresponds to the newreading position.
 10. The method according to claim 1 wherein the methodfurther comprises receiving control data from the remote user deviceindicative of a change in reading speed of the user to a new userreading speed, and modifying the streaming of the compiled soundtrackbased on the received control data.
 11. The method according to claim 10wherein the compiled soundtrack is a pre-compiled soundtrack, andwherein modifying the streaming of the compiled soundtrack based on anew user reading speed comprises selecting a new pre-compiled soundtrackhaving a reading speed that more closely matches the new user readingspeed and commencing streaming of the new pre-compiled soundtrack. 12.The method according to claim 11 wherein the method further comprisesinitiating a transition protocol for transitioning the streaming fromthe current pre-compiled soundtrack to the new selected pre-compiledsoundtrack.
 13. The method according to claim 12 wherein the transitionprotocol comprises receiving control data from the remote user deviceindicative of the user's current playback position in the streamedsoundtrack and commencing streaming of the new selected pre-compiledsoundtrack at a data position corresponding to the user's currentplayback position.
 14. The method according to claim 12 wherein thetransition protocol comprises analyzing the remaining unstreamed portionof the current pre-compiled soundtrack and identifying a transitionposition based on transition criteria, and commencing streaming of thenew selected pre-compiled soundtrack at a data position corresponding tothe identified transition position.
 15. The method according to claim 10wherein the compiled soundtrack is a real-time compiled soundtrack, andwherein modifying the streaming of the compiled soundtrack based on anew user reading position comprises commencing compiling of thesoundtrack at the new user reading speed and commencing streaming of thenew compiled soundtrack.
 16. The method according to claim 15 whereinthe method further comprises initiating a transition protocol fortransitioning the streaming from the current real-time compiledsoundtrack to the new real-time compiled soundtrack.
 17. The methodaccording to claim 16 wherein the transition protocol comprisesreceiving control data from the remote user device indicative of theuser's current playback position in the streamed soundtrack andcommencing streaming of the new real-time compiled soundtrack at a dataposition corresponding to the user's current playback position.
 18. Themethod according to claim 16 wherein the transition protocol comprisesanalyzing the remaining unstreamed portion of the current real-timecompiled soundtrack and identifying a transition position based ontransition criteria, and commencing streaming of the new real-timecompiled soundtrack at a data position corresponding to the identifiedtransition position.
 19. The method according to claim 1 wherein themethod comprises streaming the compiled soundtrack as a single digitalaudio file.
 20. The method according to claim 1 wherein the methodcomprises streaming the compiled soundtrack as a plurality of digitalaudio files, for concurrent playback at the remote user device.
 21. Themethod according to claim 20 wherein the audio regions of the soundtrackare categorised into different types and the compiled soundtrack isprovided as separate audio files, each audio file comprising the audioregions associated with one type.
 22. A soundtrack server for streaminga compiled soundtrack to a remote user device for a reader of electronicmedia content, the soundtrack defined by multiple audio regions, eachaudio region defined by: an audio track for playback in the audioregion, a start position in the electronic media content correspondingto where the playback of the audio region is to begin, and a stopposition in the electronic media content corresponding to where theplayback of the audio region is to cease, the soundtrack server beingconfigured to: receive from a remote user device over a network controldata indicative of the user's reading speed; and stream a compiledsoundtrack to the user device over the network based on the receivedcontrol data.
 23. A method of playing a compiled soundtrack on a userdevice for a reader of electronic media content, the soundtrack definedby multiple audio regions, each audio region defined by: an audio trackfor playback in the audio region, a start position in the electronicmedia content corresponding to where the playback of the audio region isto begin, and a stop position in the electronic media contentcorresponding to where the playback of the audio region is to cease, themethod comprising: sending control data to a remote server over anetwork indicative of the user's reading speed; receiving an incomingstream of a compiled soundtrack from the remote server over the network,the compiled soundtrack having an associated reading speed thatcorresponds to the sent control data; processing the streamed compiledsoundtrack and commencing playback of the soundtrack on the user device.